CM 
CD 



(19) 




? 9 

^it .** -* <*v ~* 'i 




(12) 



EuropSisches Patentamt 
European Patent Office 
Office europ6en des brevets (11) EP 0 929 1 92 A2 

EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

14.07.1999 Bulletin 1999/28 

(21) Application number: 99100184.3 

(22) Date of filing: 07.01 .1 999 




(51) IntCI. 6 : H04N5/14 



(84) 


Designated Contracting States: 


• Nakaya, Hideo 




AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 


Shinagawa-ku, Tokyo (JP) 




MC NL PT SE 


• Tanaka, Kenji 




Designated Extension States: 


Shinagawa-ku, Tokyo (JP) 




AL LT LV MK RO SI 


(74) Representative: 


(30) 


Priority: 07.01.1998 JP 1345698 


Melzer, Wolfgang, Dipl.-lng. et al 




Patentanwdlte 


(71) 


Applicant: SONY CORPORATION 


Mitschertich & Partner, 




Tokyo (JP) 


Sonnenstrasse 33 






80331 MOnchen (DE) 


(72) 


Inventors: 


• 


Kondo, Tetsujiro 
Shinagawa-ku, Tokyo (JP) 





(54) Image processing method and apparatus and recording medium therefor 



(57) An estimation section (4) calculates the motion 
vector of each pixel and its reliability from two continu- 
ous frames, and generates first-order motion distribu- 
tion images corresponding to the k-th and (k + 1)-th 
frames with any motion vector and reliability being 
assigned to each pixel. An update section (5) combines 
the input first -order motion distribution image corre- 
sponding to the (k + 1)-th frame and the fourth-order 
motion distribution image corresponding to the (k + 1)-th 
frame generated from the first -order motion distribution 
image corresponding to the k-th image to generate the 



second-order motion distribution image corresponding 
to the (k + 1 )-th frame. A prediction section (6) gener- 
ates the third-order motion distribution image corre- 
sponding to the (k + 2)-th frame according to the input 
second-order motion distribution image corresponding 
to the (k + 1)-th frame. An interpolation section (7) inter- 
polates into a blank space of the third-order motion dis- 
tribution image corresponding to the (k + 2)-th frame 
with the use of unbiased estimate to generate the 
fourth-order motion distribution image corresponding to 
the (k + 2)-th frame. 
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CROSS-REFERENCE TO RELATED APPLICATIONS 

5 [0001] The present application is related to U.S. Patent Application No. 08/816,867, filed March 13, 1997. which is 
assigned, in part, to the assignee of the present invention, and to U.S. Patent Application No. 08/837,491, filed April 18, 
1997, which is assigned to the assignee of the present invention, and both of said related U.S. Patent Applications are 
incorporated herein by reference. 

w BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0002] The present invention relates to image processing apparatuses, image processing methods, and recording 
is media therefor, and more particularly, to an image processing apparatus, an image processing method, and a recording 
medium therefor, in which motion can be estimated in units of pixels from images continuous in time. 

2. Description of the Related Art 

20 [0003] As a method for estimating a parameter which indicates an image motion (hereinafter called motion estimation) 
from a motion picture, there has been known a method using a minimum predictive residual rule, for example, a block 
matching method. In the motion-estimation method using the minimum predictive residual rule, since an image is 
divided into blocks formed of a plurality of pixels and motion estimation is performed in units of blocks, the spatial res- 
olution of motion estimation is low and motion estimation can be performed only for a rigid body whose constituent parts 

25 move in parallel. As a result, motion estimation cannot be performed using this method for a nonrigid body having une- 
ven motion distribution. 

[0004] In the motion-estimation method using the minimum predictive residual rule, for example, with the block match- 
ing method, when an actual movement is out of a search range, motion estimation has a large error, and a low robust- 
ness appears. Accordingly, it would be desirable to provide a method, apparatus and computer program for increasing 
30 spatial resolution in motion estimation of an image and to ensure robustness. The present invention has been made in 
consideration of the above conditions. 

BRIEF SUMMARY OF THE INVENTION 

35 [0005] It is an object of the present invention to increase the spatial resolution of motion estimation and to ensure 
robustness by a non-decision-theory method with the use of unbiased estimate. 

[0006] It is a feature of the present invention to perform motion estimation in units of pixels by the use of unbiased 
estimation, thereby increasing the spatial resolution of motion estimation and robustness. 

[0007] It is an advantage of the present invention to increase spatial resolution in the motion estimation of an image. 

40 [0008] According to one aspect of the invention, a method for estimating motion of a motion picture, the motion picture 
comprising a plurality of images includes the steps of estimating from a k-th image and a (k + 1)-th image a motion 
parameter corresponding to each pixel of the k-th image to generate a first-order motion distribution image correspond- 
ing to the k-th image, the first-order motion distribution image corresponding to the k-th image represented by the 
motion parameters corresponding to the pixels of the k-th image; estimating from at least the (k + 1)-th image a motion 

45 parameter corresponding to each pixel of the (k + 1 )-th image to generate a first-order motion distribution image corre- 
sponding to the (k + 1)-th image, the first-order motion distribution image corresponding to the (k + 1)-th image repre- 
sented by the motion parameters corresponding to the pixels of the (k + 1)-th image; updating the first-order motion 
distribution image corresponding to the (k + 1)-th image to generate a second-order motion distribution image corre- 
sponding to the (k + 1)-th image; predicting, from the image second-order motion distribution image corresponding to 

50 the (k + 1 )-th image, a third-order motion distribution image corresponding to the (k + 2)-th image; and interpolating the 
third-order motion distribution image corresponding to the (k + 2)-th image to generate a fourth-order motion distribution 
image corresponding to the (k + 2)-th image, where k is an integer, and wherein the step of updating updates the first- 
order motion image distribution image corresponding to the (k + 1)-th image using a higher than first-order motion dis- 
tribution image generated from the k-th image. 

55 [0009] According to a further aspect of the present invention, an apparatus for estimating motion of a motion picture, 
the motion picture comprising a plurality of images includes means for estimating from a k-th image and a (k + 1)-th 
image a motion parameter corresponding to each pixel of the k-th image to generate a first-order motion distribution 
image corresponding to the k-th image, the first-order motion distribution image corresponding to the k-th image repre- 
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sented by the motion parameters corresponding to the pixels of the k-th image; means for estimating from at feast the 
(k + 1 )-th image a motion parameter corresponding to each pixel of the (k + 1)-th image to generate a first-order motion 
distribution image corresponding to the (k + 1)-th image, the first-order motion distribution image corresponding to the 
(k ■+ 1)-th image represented by the motion parameters corresponding to the pixels of the (k + 1)~th image; means for 

5 updating the first-order motion distribution image corresponding to the (k + 1)-th image to generate a second-order 
motion distribution image corresponding to the (k + 1)-th image; means for predicting, from the image second-order 
motion distribution image corresponding to the (k + 1)-th image, a third-order motion distribution image corresponding 
to the (k + 2)-th image; and means for interpolating the third-order motion distribution image corresponding to the (k + 
2)-th image to generate a fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is an 

w integer, and wherein the means for updating updates the first-order motion image distribution image corresponding to 
the (k + 1)-th image using a higher than first-order motion distribution image generated from the k-th image. 
[0010] According to another aspect of the present invention, an apparatus for estimating motion of a motion picture, 
where the motion picture comprising a plurality of images, includes a circuit for estimating from a k-th image and a (k + 
1)-th image a motion parameter corresponding to each pixel of the k-th image to generate a first-order motion distribu- 

is tion image corresponding to the k-th image, the first-order motion distribution image corresponding to the k-th image 
represented by the motion parameters corresponding to the pixels of the k-th image; a circuit for estimating from at least 
the (k + 1)-th image a motion parameter corresponding to each pixel of the (k + 1)-th image to generate a first-order 
motion distribution image corresponding to the (k + 1)-th image, the first-order motion distribution image corresponding 
to the (k + 1 )-th image represented by the motion parameters corresponding to the pixels of the (k + 1)-th image; a cir- 

20 cuit for updating the first-order motion distribution image corresponding to the (k + 1)-th image to generate a second- 
order motion distribution image corresponding to the (k + 1)-th image; a circuit for predicting, from the image second- 
order motion distribution image corresponding to the (k + 1)-th image, a third-order motion distribution image corre- 
sponding to the (k + 2)-th image; and a circuit for interpolating the third-order motion distribution image corresponding 
to the (k + 2)-th image to generate a fourth-order motion distribution image corresponding to the (k + 2)-th image, where 

25 k is an integer, and wherein the circuit for updating updates the first-order motion image distribution image correspond- 
ing to the (k + 1)-th image using a higher than first-order motion distribution image generated from the k-th image. 
[0011] According to yet a further aspect of the present invention, an apparatus for estimating motion of a motion pic- 
ture, the motion picture comprising a plurality of images, includes a first estimator for estimating from a k-th image and 
a (k + 1)-th image a motion parameter corresponding to each pixel of the k-th image to generate a first-order motion 

30 distribution image corresponding to the k-th image, the first-order motion distribution image corresponding to the k-th 
image represented by the motion parameters corresponding to the pixels of the k-th image; a second estimator for esti- 
mating from at least the (k + 1 )-th image a motion parameter corresponding to each pixel of the (k + 1 )-th image to gen- 
erate a first-order motion distribution image corresponding to the (k + 1)-th image, the first-order motion distribution 
image corresponding to the (k + 1)-th image represented by the motion parameters corresponding to the pixels of the 

35 (k + 1)-th image; an updater for updating the first-order motion distribution image corresponding to the (k + 1)-th image 
to generate a second-order motion distribution image corresponding to the (k + 1)-th image; a predictor for predicting, 
from the image second-order motion distribution image corresponding to the (k + 1)-th image, a third-order motion dis- 
tribution image corresponding to the (k + 2)-th image; and an interpolator for interpolating the third-order motion distri- 
bution image corresponding to the (k + 2)-th image to generate a fourth-order motion distribution image corresponding 

40 to the (k + 2)-th image, where k is an integer, and wherein the circuit for updating updates the first-order motion image 
distribution image corresponding to the (k + 1)-th image using a higher than first-order motion distribution image gen- 
erated from the k-th image. 

[001 2] According to yet another aspect of the present invention, a program storage device readable by a machine, 
tangibly embodies a program of instructions executable by the machine to perform method steps for estimating motion 

45 of a motion picture, where the steps include estimating from a k-th image and a (k + 1)-th image a motion parameter 
corresponding to each pixel of the k-th image to generate a first-order motion distribution image corresponcfing to the 
k-th image, the first-order motion distribution image corresponding to the k-th image represented by the motion param- 
eters corresponding to the pixels of the k-th image; estimating from at least the (k + 1)-th image a motion parameter 
corresponding to each pixel of the (k + 1 )-th image to generate a first-order motion distribution image corresponding to 

so the (k + 1)-th image, the first-order motion distribution image corresponding to the (k + 1)-th image represented by the 
motion parameters corresponding to the pixels of the (k + 1 )-th image; updating the first-order motion distribution image 
corresponding to the (k + 1)-th image to generate a second-order motion distribution image corresponding to the (k + 
1)-th image; predicting, from the image second-order motion distribution image corresponding to the (k + 1)-th image, 
a third-order motion distribution image corresponding to the (k + 2)-th image; and interpolating the third-order motion 

55 distribution image corresponding to the (k + 2)-th image to generate a fourth-order motion distribution image corre- 
sponding to the (k + 2)-th image, where k is an integer, and wherein the step of updating updates the first-order motion 
image distribution image corresponding to the (k + 1)-th image using a higher than first-order motion distribution image 
generated from the k-th image. 
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[0013] These and other objects, features and advantages will become apparent when considered with reference to 
the following description and the accompanying drawings, where like reference numerals refer to the same or similar 
elements. 

BRIEF DESCRIPTION OF THE OF THE DRAWINGS 
[0014] 

Fig. 1 is a block diagram showing a configuration of an image processing apparatus to which the present invention 
is applied. 

Fig. 2 is a flowchart illustrating operations of the image processing apparatus shown in Fig. 1 . 

Fig. 3 is a view showing operations of the image processing apparatus shown in Fig. 1 . 

Fig. 4 is a flowchart of estimation processing shown in a step S3 of Fig. 2. 

Fig. 5 is a view illustrating prediction processing shown in a step S6 of Fig. 2. 

Fig. 6 is another view illustrating the prediction processing shown in the step S6 of Fig. 2. 

Fig. 7 is a flowchart of the prediction processing shown in the step S6 of Fig. 2. 

Fig. 8 is a flowchart of interpolation processing shown in a step S7 of Fig. 2. 

Fig. 9 is a flowchart of update processing shown in a step S5 of Fig. 2. 

Figs. 10A and 10B are photographs of gray-scale images shown on a display for comparison between the present 
invention and conventional image processing. 

Figs. 1 1 A, 1 1 B, 1 1 C and 1 1 D show photos of gray-scale images shown on a display to describe a case in which 
the present invention is applied to a zoom-out image. 

DETAILED DESCRIPTION OF THE INVENTION 

[001 5] An embodiment of the present invention is now described below. A configuration of an image processing appa- 
ratus to which the present invention is applied will be described below by referring to Fig. 1. A control section 1 in this 
image processing apparatus controls the whole apparatus. A hard disk 2 stores motion pictures input in advance, and 
outputs the motion pictures to a reproduction section 3 according to control by the control section 1. A series of motion 
pictures (one scene) is formed of a plurality of still images (frames) to which serial frame numbers (fr = 0, 1, 2, ...k, k + 
1 N) are assigned. 

[001 6] The reproduction section 3 reproduces the motion pictures sent from the hard disk 2, in units of frames (as still 
images) and outputs such data to an estimation section 4. 

[0017] The estimation section 4 calculates a motion vector for each pixel and its reliability corresponding to the k-th 
frame from the k-th and (k + l)-th frames which are continuous in time order and sent from the reproduction section 3. 
The estimation section 4 also generates the first-order motion distribution image corresponding to the k-th frame by 
assigning a motion vector and its reliability to each pixel (performing unbiased estimation), and outputs such data to an 
update section 5. The first-order motion distribution image corresponding to the 0th frame is outputted to a prediction 
section 6. 

[001 8] The update section 5 combines the first-order motion distribution image corresponding to the (k + 1 )-th frame 
sent from the estimation section 4 and the fourth-order motion distribution image corresponding to the (k + 1)-th frame 
sent from an interpolation section 7 (described later) to generate the second-order motion distribution image corre- 
sponding to the (k + 1 )-th frame and outputs it to the prediction section 6. In the case of the first-order motion distribution 
image corresponding to the 0th frame (fr = 0), this update operation is not performed. 

[001 9] The prediction section 6 predicts the third-order motion distribution image corresponding to the (k + 1 )-th frame 
according to the second-order motion distribution image corresponding to the k-th frame sent from the update section 
5 (or in the case of fr = 0. the first-order motion distribution image corresponding to the 0th frame sent from the estima- 
tion section 4) and outputs it to the interpolation section 7. In this case, the motion vector for each pixel in the third-order 
motion distribution image is predicted with an assumption that the motion vector for each pixel in the second-order dis- 
tribution image (or in the case of fr = 0, the first-order motion distribution image corresponding to the 0th frame) does 
not change. 

[0020] The interpolation section 7 interpolates into a blank space of the third-order motion distribution image corre- 
sponding to the input (k + 1)-th frame with the use of an unbiased estimate to generate the fourth-order motion distri- 
bution image corresponding to the (k + 1)-th frame. The fourth-order motion distribution image corresponding to the (k 
+ 1)-th frame is outputted as a final motion distribution image representing motion from the k-th frame to the (k + 1)-th 
frame and is provided to the update section 5. 

[0021] A memory 8 temporarily stores information as a work area of each section and outputs it to each section. As 
shown in Fig. 1 , the control section 1 , hard disk 2, reproduction section 3, estimation section 4, update section 5, pre- 
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diction section 6, interpolation section 7 and memory 8 are coupled via a bus 9. 

[0022] An operation of this image processing apparatus will be described below by referring to a f bwchart shown in 
Fig. 2 and Fig. 3. In a step S1 . the control section 1 checks for the total number N of frames constituting one scene 
stored in the hard disk 2, and initializes a parameter fr indicating the order of the frames (fr is set to zero). 
5 [0023] In a step S2, it is determined whether or not fr is N (N indicates the last frame). When it is determined that fr 
is not N, the processing proceeds to a step S3. 

[0024] In the step S3, estimation processing is executed. Details of the estimation processing will be described below 
by referring to the flowchart of Fig. 4. 

[0025] In a step S1 1 , the coordinates (i, j) of the pixel to be estimated in an estimated frame (the k-th frame) are ini- 
10 tialized to (0. 0). In a step S12, it is determined whether the coordinates (i, j) of the pixel to be estimated are not larger 
than the maximum (i^x, j max ) (that is, within a frame). When it is determined that the coordinates (i, j) of the pixel to be 
estimated are not larger than the maximum (imax, jmax) (that is, within the frame), the processing proceeds to a step 
S13. 

[0026] In the step S13. variables "counter" and "sum" are set to zero. In a step S14, the coordinates (ix, iy) of a refer- 
is enced pixel in the referenced frame (the (k + 1)-th frame) are initialized to (i-range, j-range). The referenced pixel area 
in the reference frame is a square having a side with the specified length, a 2 x range, and having a center at the coor- 
dinates (i, j). Therefore, the coordinates (ix-range, iy-range) indicate the pixel at the upper left corner of the referenced 
area. 

[0027] In a step S15, the value of 1/(the sum of the squares of the difference between the pixel values of the pixel (i, 
20 j) to be estimated and the referenced pixel (ix, iy) + 0.01) is calculated and added to the variable "sum." The value 0.01 
is a compensation value for preventing the reliability from diverging to infinite. 

[0028] In a step S16, the differences (ix-i, iy-j) (motion vector) between the coordinates (ix, iy) of the reference pixel 
and the coordinates (i,j) of the pixel to be estimated are calculated. The calculated values (motion vector) are disposed 
in an array iv_hist[counter] with the serial number "counter" of the pixel in the referenced area. In the same way, the var- 
25 table "sum" is disposed in an array sum_hist[counter] with the serial number "counter" of the pixel in the referenced 
area. The variable "sum" and the array sum_hist[counter] are used for calculating reliability p in a step S21 described 
later. 

[0029] In a step SI 7, it is determined whether the coordinates (ix, iy) of the referenced pixel are smaller than the max- 
imum (i + range, j + range) (whether a pixel which has not yet been referenced exists within the referenced area). When 

30 it is determined that they are not smaller (that all the pixels in the referenced area have been referenced), the process- 
ing proceeds to a step S18. If a pixel which has not yet been referenced does exist, the processing proceeds to a step 
S23 and the variable "counter" is incremented. The processing then proceeds to a step S24, where the coordinates (ix, 
iy) are incremented. The processing of the step SI 5 is then executed with another pixel being set to be handled. 
[0030] In the step S18, a variable "sum_thres" is defined as the variable "sum" multiplied by a random number 

35 (between 0 and 1). In other words, a threshold is defined with the use of a random number. In a step S19, the variable 
"counter" is initialized to zero. 

[0031] In a step S20, it is determined whether the value of the array sum_hist[counter] is greater than sum_thres 
(threshold) defined at random in the step S18 (whether it is inappropriate as a referenced pixel). If the value of the array 
sum_hist(counter] is less than or equal to sum_thres (it is inappropriate as a referenced pixel), the processing proceeds 
40 to a step S25. The variable "counter" is incremented and the processing returns to the step S20. As described in the 
step S15. since processed values are sequentially accumulated in the variable "sum," when the variable "counter" 
becomes large, the value of sum_hist[counter] also becomes large. As a result, a possibility that the value of 
sum_hist[counter] becomes larger than the value of sum_thres increases. 

[0032] When it is determined that the value of the array sum_hist[counter] is greater than sum__thres (that is, the pixel 
45 is appropriate as a referenced pixel), the processing proceeds to a step S21. In the step S21, the motion vector dis- 
posed in the array "iv_ hist[counter]" corresponding to the array "sum-hist [counter}" is copied as the motion vector of 
the pixel to be estimated (the pixel at the coordinates (i, j)), and the value of (sum_hist[counter] - sum_hist(counter - 
1])/sum is calculated as its reliability p and set. 

[0033] In a step S22, the coordinates (i, j) are incremented. The processing returns to the step S12 with other coor- 
50 dinates being set to be processed. 

[0034] In the step S12, if it is determined that (i, j) are larger than the maximum (i max , j max ) (not within the frame), the 
estimation processing for the first-order estimation distribution image corresponding to the fr-th frame is finished and 
the processing for the first-order estimation distribution image corresponding to the fr-th frame returns to the step S3 in 
Fig. 2. 

55 [0035] In the foregoing way, the first-order motion distribution image for fr = 0 shown in Fig. 3 has, for example, been 
generated with the use of unbiased estimate. 
[0036] The flowchart shown in Fig. 2 will be again described. 

[0037] After the estimation processing has been performed in the foregoing way, it is determined in a step S4 whether 
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the frame parameter fr = 0 (indicates the start frame). When it is determined that fr is not zero, the processing proceeds 
to a step S5. In the step S5, the update section 5 executes update processing, tf, however, fr equal to is zero at first, the 
update processing in the step S5 is skipped and the processing proceeds to a step S6. 

[0038] In the step S6 t prediction processing is executed as shown in Fig. 5. Namely, a motion vector for each pixel 
corresponding to the (fr + 1)-th frame is predicted with an assumption that pixels move at the same speed in the input 
second-order motion distribution image corresponding to the fr-th frame (or the first-order motion distribution image cor- 
responding to the Oth frame if f r is zero). Also, as shown in Fig. 6, when a plurality of motion vectors can be considered, 
the motion vector having the maximum reliability p is employed. Details of the prediction processing will be described 
below by referring to the flowchart of Fig. 7. 

[0039] In a step S3t, the motion vector (x, y) and the reliability p of the input second-order motion distribution image 
corresponding to the fr-th frame (or the first-order motion distribution image corresponding to the fr-th frame if fr is zero) 
are arranged in an input data array src[i, j]. 

[0040] In a step S32, an output array dst[i, j] is initialized (motion vectors and reliabilities p are all set to zero). In a 
step S33, the coordinates (i. j) of the pixel to be predicted are initialized to (0, 0). 

[0041] In a step S34, it is determined whether the coordinates (i, j) of the pixel to be predicted of the fr-th frame are 
not larger than the maximum (imax, jmax) (within the frame). If it is determined that the coordinates (i, j) are not larger 
than the maximum (imax, jmax) (within the frame), the processing proceeds to a step S35. 

[0042] In the step S35, variables (fx, fy) are set to the motion vector (x, y) of the input data array srcp, j] (fx is set to 
the x component of the motion vector, and fy is set to the y component of the motion vector). 

[0043] In a step S36, it is determined whether the reliability p of dst[i + fx, j + fy] is smaller than that of srcp, j]. When 
it is determined that the reliability p of dstp + fx, j + fy] is smaller than that of srcp, j], the processing proceeds to a step 
S37. In the step S37, the motion vector of dstp + fx, j + fy] (motion vector of the third-order motion distribution image) is 
set to that of srcp, j]. In other words, the processing shown in Fig. 6 is executed. When it is determined that the reliability 
p of dstp + fx, j + fy] is not smaller than that of srcp, j], or when the step S37 has been finished, the coordinates (i, j) are 
incremented in step S38 and the processing then returns to the step S34 with another pixel being set to be handled. 
[0044] In the step S34, when it is determined that the coordinates (i. j) of the pixel to be predicted are larger than the 
maximum (imax, jmax) (not within the frame), the prediction processing is finished and the processing returns to the 
step S6 in Fig. 2. In the foregoing way, a third-order motion distribution image corresponding to the (fr + 1)-th frame is 
obtained by the prediction processing shown in Fig. 3. 
[0045] The processing shown in Fig. 2 will again be described. 

[0046] After the prediction processing for the third-order motion distribution image corresponding to the (fr + 1)-th 
frame has been executed in the foregoing way, the interpolation section 7 executes interpolation processing to the third- 
order motion distribution image corresponding to the (fr + 1)-th frame in a step S7. Details of the interpolation process- 
ing will be described below by referring to a flowchart shown in Fig. 8. 

[0047] In a step S41, the coordinates (i, j) of a pixel of the (fr + 1)-th frame which is to be interpolated (interpolation 
target) is initialized to (0, 0). In a step S42, it is determined whether the coordinates (i, j) are not larger than the maxi- 
mum ("max. jmax) (are within a frame). When it is determined that the coordinates (i, j) of the pixel to be interpolated are 
not larger than the maximum (imax, jmax) (are within a frame), the processing proceeds to a step S43. 
[0048] In the step S43, the interpolation section 7 determines whether the reliability p(i, j) of the pixel at the coordi- 
nates (i, j) is less than the specified reference value. When it is determined that the reliability p is less than the specified 
reference value (the pixel lacks information), the processing proceeds to a step S44. When the reliability p(i, j) is not 
smaller than the specified value, since the pixel does not lack information, interpolation processing is unnecessary. The 
processing proceeds to a step S55. The coordinates (i, j) are incremented. The processing then returns to the step S42 
with another pixel being set to be handled, and the same processing is executed. 

[0049] In the step S44, variables "counter" and "sum" are set to zero. In a step S45, the coordinates (ix, iy) of a refer- 
enced pixel in the reference frame (the (fr + 1)-th frame) are initialized to (i-range, j-range). The referenced pixel area is 
a square having a side with the specified length, 2 x range, and having a center at the coordinates (i, j). Therefore, the 
coordinates (i-range, j-range) indicate the pixel at the upper left corner of the referenced area. 
[0050] In a step S46, the interpolation section 7 determines whether the reliability p(ix, iy) of the pixel at the coordi- 
nates (ix. iy) is larger than zero. When it is determined that the reliability is larger than zero (that the pixel is appropriate 
as the source pixel for copying its information (pixel value) onto a pixel which lacks information), the processing pro- 
ceeds to a step S47. When the reliability p is zero, the pixel is inappropriate as the copy-source pixel. The processing 
therefore proceeds to a step S56. The coordinates (ix, iy) are incremented and the processing returns to the step S46 
with another pixel being set to be handled. 

[0051 ] In step S47, the value of L((ix, iy), (i, j)) is added to the variable "sum," where L((ix, iy), (i, j)) indicates a function 
which shows whether the pixel at the coordinates (ix, iy) is appropriate as the copy-source pixel for the pixel (i, j) to be 
interpolated and is expressed as follows: 
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L((ix, iy),(i,j)) 
= pCix, iy) x Exp(-D1 ((ix, iy), (i, j))/a - D2((ix. iy), (i. j)/b) 

5 where p(ix, iy) indicates the reliability of the referenced pixel, D1 ((ix, iy), (i, j)) indicates the distance between the coor- 
dinates of (ix, iy) and (i, j), and D2((ix, iy), (i, j)) indicates the difference between the pixel values of (ix, iy) and (i, j). 
[0052] In a step S48, the differences (ix-i, iy-j) (motion vector) between the coordinates (ix. iy) of the referenced pixel, 
which is regarded as a copy-source pixel whose pixel value is appropriate, and the coordinates (i, j) of the pixel to be 
interpolated are calculated. The calculated values (motion vector) are disposed in an array iv_hist[counter] with the 

10 serial number "counter" of the pixel in the referenced area. In the same way the variable "sum" is disposed in an array 
sum_hist[counter] with the serial number "counter" of the pixel in the referenced area. 

[0053] In a step S49, it is determined whether the coordinates (ix. iy) of the referenced pixel are smaller than the max- 
imum (i + range, j + range) (whether a pixel which has not yet been referenced exists within the referenced area). When 
it is determined that they are not smaller (that all the pixels in the referenced area have been referenced), the process- 
is ing proceeds to a step S50. If a pixel which has not yet been referenced remains, the processing proceeds to a step 
S57 and the variable "counter" is incremented. The processing then proceeds to a step S56. The coordinates (ix, iy) are 
incremented. The processing from the step S46 is executed with another pixel being set to be handled. 
[0054] In a step S50, it is determined whether the variables "sum" and "counter" are larger than zero. When it is deter- 
mined that both "sum" and "counter" are larger than zero (a pixel appropriate for a copy source exists in the referenced 
20 area), the processing proceeds to a step S51 . When at least one of "sum" and "counter" is zero, since a pixel appropri- 
ate for a copy source does not exist in the referenced area, the processing proceeds to a step 58. The reliability p of the 
coordinates (ix, iy) is set to zero, and the processing proceeds to the step S55. The coordinates (ix, iy) are incremented, 
and the processing from the step S42 is executed with another pixel being set to be handled. 

[0055] In the step S51, a variable "sumjthres" is defined as the variable "sum" multiplied by a random number 
25 (between 0 and 1). In other words, a threshold is defined with the use of a random number. In a step S52. the variable 
"counter is initialized to zero. 

[0056] In a step S53, it is determined whether the value of the array sum_hist[counter] is greater than sum_thres 
(threshold) defined at random in the step 51 (whether it is inappropriate as a copy-source pixel). If the value of the array 
sum_hist[counter] is not greater than sum_thres (it is inappropriate as a copy-source pixel), the processing proceeds to 
30 a step S59. The variable "counter" is incremented and the processing returns to the step S53. As described in the step 
S47, since processed values are sequentially accumulated in the variable "sum," when the variable "counter" becomes 
large, the value of sum_hist[counter] also becomes large. As a result, a possibility that the value of sumjiistjcounter] 
becomes larger than the value of sum_thres increases. 

[0057] When the value of the array sum_hist[counter] is greater than surnjhres (it is appropriate as a copy-source 
35 pixel), the processing proceeds to a step S54. In the step S54, the reliability and the motion vector disposed in the array 
"iv_hist[counter]" are copied as the reliability p(i, j) and the motion vector V1 (i, j) of the pixel to be interpolated. Then, in 
the step S55, the coordinates (i, j) are incremented. The processing returns to the step S42 and the subsequent 
processing is executed. 

[0058] In the step S42, if it is determined that (i. j) are larger than the maximum (imax, jmax) (not within the frame) (in 
40 other words, that all pixels in the frame have been processed), the interpolation processing is finished and the process- 
ing returns to the step S7 in Fig. 2. 

[0059] In the foregoing way, a fourth-order motion distribution image corresponding to the (fr + 1)-th frame has been 
generated by the interpolation processing with the use of an unbiased estimate as shown in Fig. 3. The fourth-order 
motion distribution image corresponding to the (fr + 1)-th frame is outputted as the final motion distribution image and 
45 is used to update the first-order distribution image corresponding to the (fr + 1)-th frame and to generate the second- 
order motion distribution image corresponding to the (fM)-th frame. The flowchart shown in Fig. 2 will be again 
described. 

[0060] After the interpolation processing has been performed in the foregoing way, fr is incremented by 1 in a step S8, 
and the processing returns to the step S2. 
so [0061 ] In the steps S2 and S3, the above processing is executed in the same way. In the step S4. since fr is not zero, 
the processing proceeds to the step S5, and the update section 5 executes update processing. Details of the update 
processing will be described below by referring to Fig. 9. 

[0062] In a step S71 , the coordinates (i, j) of the pixel to be updated is initialized to (0, 0). In a step S72, it is deter- 
mined whether (i, j) are not larger than the maximum (imax, jmax) (are within a frame). When it is determined that (i, j) 
55 are not larger than the maximum (imax, jmax) (are within the frame), the processing proceeds to a step S73. 

[0063] In the step S73, a variable pp is set to the reliability p of a data array prep, j] of the fourth-order motion distri- 
bution image corresponding to the still image having an fr of (k + 1) (initially k = 0) sent from the interpolation section 7. 
In the same way, a variable cp is set to the reliability p of a data array curfi, j] of the first-order motion distribution image 
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corresponding to the still image having an fr of (k + 1) sent from the interpolation section 7. 

[0064] In a step S74, it is determined whether the variable pp is greater than the variable cp. When it is determined 
that pp is greater than cp (the reliability p of the fourth-order motion distribution image is higher than that of the first- 
order motion distribution image), the processing proceeds to a step S75. When it is determined that pp is not greater 

5 than cp (the reliability p of the first-order motion distribution image is higher than that of the fourth-order motion distri- 
bution image), the motion vector is not updated and the processing proceeds to a step S76. The coordinates (i, j) are 
incremented and the processing returns to the step S72 with another pixel being set to be handled. 
[0065] In the step S75 f the weighted average of the motion vector of the first-order motion distribution image and that 
of the fourth-order motion distribution image are calculated from the following expression with the use of "pp" and "cp" 

10 to obtain and set the motion vector of the second-order motion distribution image corresponding to the (k + 1)-th frame. 

Motion vector of second-order motion distribution image 
« (motion vector of prep, j]) x {pp/(pp + cp)} 
+ (motion vector of curp, j]) x {cp/(pp + cp)} 

[0066] In the step S72, when it is determined that (i, j) are larger than the maximum (imax, jmax). the update process- 
ing is finished and the processing returns to the step S5 in Fig. 2. 

[0067] In the foregoing way, the second-order motion distribution image is obtained by the update processing as 
20 shown in Fig. 3. 

[0068] Motion distribution images obtained by the above image processing apparatus will now be compared with 
those obtained by the conventional method. Fig. 10A shows motion distribution images obtained by an image process- 
ing apparatus to which the present invention is applied. Fig. 10B shows motion distribution images obtained according 
to the maximum likelihood estimation, which is a conventional method. The same images (in which a model truck moves 

25 in the left direction on a fixed table) are handled in both methods. In Fig. 10A, starting from the uppermost image, there 
are shown the first-order motion distribution images corresponding to the still image having an fr = 0. the third-order 
motion distribution images corresponding to the still image having an fr = 1 , and a third-order motion distribution images 
corresponding to the still image having an fr = 4. The density of a pixel in these images indicates the magnitude of the 
motion vector in the horizontal direction (in the left direction in the images). 

30 [0069] In the images shown in Fig. 10B, the contour of the model truck is dim and pixels of a table, which is fixed and 
below the model, have a certain density (which is indicative of movement). On the other hand, in the images shown in 
Fig. 10A, information is accumulated as fr increases. The contour of the model truck therefore becomes clear and the 
density of pixels of the fixed portion (table) decreases. Stated differently, the relative contrast increases as between the 
fixed portion of the image and the moving portion of the image. Therefore, it is determined that Fig. 10A shows motion 

35 distribution images having a less erroneous decision. 

[0070] With a correct motion distribution image thus being obtained, an image of a moving portion can be taken out 
and traced, and motion estimation becomes possible. 

[0071 ] With reference now to Figs. 1 1 A - 1 1 D, when an image is zoomed out, it is considered the motion vector (hor- 
izontal component of the image) of a pixel has a direction toward the center of the image as shown in Fig. 1 1 D and has 

40 a magnitude which increases as its position becomes away from the center line. Fig. 1 1 A illustrates an image based 
upon source data representative of the image of butterflies. Fig 1 1 B show a zoom out of the image of Fig. 1 1 A, where 
such Fig. 11 B is a third-order motion distribution image corresponding to f r = 1 . In Fig. 11 C, which shows the third-order 
motion distribution image corresponding to an fr = 4, the density of a pixel in the image gradually becomes thick from 
the left end to the right end. In other words, the motion vector continuously changes its direction from negative to posi- 

45 tive when the left direction is set to positive as the position moves from the left end to the right end. Its nature is similar 
to that shown in Fig. 1 1 D. Therefore, it can be said that this image processing apparatus of the present invention can 
be utilized to can perform motion estimation for zooming. 

[0072] Although only horizontal movement is used as an example to explain the present invention, the present inven- 
tion is equally applicable to movement along other axes such as the vertical axis. Furthermore, the present invention 
so can also be applied to estimation of a parameter which indicates depth. 

[0073] Although in order to explain the present invention, the fourth-order motion distribution image is outputted as a 
final motion distribution image, another motion distribution image, that is, the first, second or third motion distribution 
image can be outputted as a final motion distribution image. 

[0074] A computer program which implements each of the above processing can be offered to the user through net- 
55 work media such as the Internet and a digital satellite. In addition, a program storage device or an information recording 
media, such as a magnetic disk, a CD-ROM, a RAM, magnetic tape or a DVD, may be utilized to embody such a com- 
puter program. Such computer program may be executed by a machine to perform method steps for estimating motion 
in accordance with the present invention. 
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[0075] As described above, according to an image processing apparatus, an image processing method and a record- 
ing medium of the present invention, since motion estimation is performed in units of pixels by use of unbiased estima- 
tion, the spatial resolution of motion estimation is increased and robustness is ensured. 

[0076] Although only certain embodiments have been described in detail, those having ordinary skill in the art will 
5 certainly understand that many modifications are possible without departing from the teachings hereof. All such modi- 
fications are intended to be encompassed within the following claims. 

Claims 

10 1 . A method for estimating motion of a motion picture, the motion picture comprising a plurality of images, the method 
comprising the steps of: 

estimating (4) from a k-th image and a (k + 1)-th image a motion parameter corresponding to each pixel of the 
k-th image to generate a first-order motion dstribution image corresponding to the k-th image, the first-order 
15 motion distribution image corresponding to the k-th image represented by the motion parameters correspond- 

ing to the pixels of the k-th image; 

estimating (4) from at least the (k + 1)-th image a motion parameter corresponding to each pixel of the (k + 1)- 
th image to generate a first-order motion distribution image corresponding to the (k + 1 )-th image, the first-order 
motion distribution image corresponding to the (k + 1)-th image represented by the motion parameters corre- 
20 I spending to the pixels of the (k + 1 )-th image; 

updating (5) the first-order motion distribution image corresponding to the (k + 1)-th image to generate a sec- 
ond-order motion distribution image corresponding to the (k + 1)-th image; 

predicting (6), from the image second-order motion distribution image corresponding to the (k + 1)-th image, a 
third-order motion distribution image corresponding to the (k + 2)-th image; and 
25 interpolating (7) the third-order motion distribution image corresponding to the (k + 2)-th image to generate a 

fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is an integer, and 
wherein the step of updating updates the first-order motion image distribution image corresponding to the (k + 
1)-th image using a higher than first-order motion distribution image generated from the k-th image. 

30 2. The method according to claim 1 , wherein the higher than first-order motion distribution image generated from the 
k-th image is a fourth-order motion distribution image corresponding to the (k + 1)-th image. 

3. The method according to claim 1 , wherein the higher than first-order motion distribution image generated from the 
k-th image is a fourth-order motion distribution image corresponding to the (k + 1)-th image, further comprising the 

35 steps of: 

predicting (6) from the first-order motion distribution image corresponding to the k-th image a third-order 
motion distribution image corresponding to the (k + 1)-th image; and 

interpolating (7) the (k + 1)-th third-order motion distribution image to generate the fourth-order motion distri- 
40 bution image corresponding to the (k + 1)-th image. 

4. The method according to anyone of the preceding claims, 

wherein the step of predicting (6) predicts a motion distribution image corresponding to the (k + 2)-th image from 
the second-order image corresponding to the (k + 1)-th image with an assumption that pixels move at a same 
45 speed as in the second-order image corresponding to the (k + 1)-th image. 

5. The method according to anyone of the preceding claims, 

wherein the motion parameter includes a motion vector and an indication of reliability. 

so 6. The method according to claim 5, 

wherein the step of updating combines, for each pixel, a motion vector representing the first-order motion distribu- 
tion image corresponding to the (k + 1)-th image and a motion vector representing a higher than first-order motion 
distribution image generated from the k-th image according to the reliability of the first-order and the higher than 
first-order motion distribution image generated from the k-th image. 

55 

7. The method according to claim 6, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 
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8. An apparatus for estimating motion of a motion picture, 

the motion picture comprising a plurality of images, the apparatus comprising: 

means (4) for estimating from a k-th image and a (k + 1)-th image a motion parameter corresponding to each 
pixel of the k-th image to generate a first-order motion distribution image corresponding to the k-th image, the 
first-order motion distribution image corresponding to the k-th image represented by the motion parameters 
corresponding to the pixels of the k-th image; 

means (4) for estimating from at least the (k + 1)-th image a motion parameter corresponding to each pixel of 
the (k + l)-th image to generate a first-order motion distribution image corresponding to the (k + i)-th image, 
the first-order motion distribution image corresponding to the (k + 1)-th image represented by the motion 
parameters corresponding to the pixels of the (k + 1 )-th image; 

means (5) for updating the first-order motion distribution image corresponding to the (k + 1 )-th image to gener- 
ate a second-order motion distribution image corresponding to the (k + 1)-th image; 

means for predicting (6), from the image second-order motion distribution image corresponding to the (k + 1)- 

th image, a third-order motion distribution image corresponding to the (k + 2)-th image; and 

means for interpolating (7) the third-order motion distribution image corresponding to the (k + 2)-th image to 

generate a fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is an integer, 

and 

wherein the means for updating updates the first-order motion image distribution image corresponding to the 
(k + 1)-th image using a higher than first-order motion distribution image generated from the k-th image. 

. 9. The apparatus according to claim 8, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 

10. The apparatus according to claim 8. 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1 )-th image, wherein the means (6) for predicting predicts from the first- 
order motion distribution image corresponding to the k-th image a third-order motion distribution image correspond- 
ing to the (k + 1)-th image; and the interpolating means (7) interpolates the (k + 1)-th third-order motion distribution 
image to generate the fourth-order motion distribution image corresponding to the (k + 1)-th image. 

1 1 . The apparatus according to anyone of claims 8 to 1 0, 

wherein the means (6) for predicting predicts a motion distribution image corresponding to the (k + 2)-th image from 
the second-order image corresponding to the (k + 1)-th image with an assumption that pixels move at a same 
speed as in the second-order image corresponding to the (k + 1)-th image. 

12. The apparatus according to anyone of claims 8 to 11, 

wherein the motion parameter includes a motion vector and an indication of reliability. 

13. The apparatus according to claim 12, 

wherein the means (5) for updating combines, for each pixel, a motion vector representing the first-order motion 
distribution image corresponding to the (k + 1)-th image and a motion vector representing a higher than first-order 
motion distribution image generated from the k-th image according to the reliability of the first-order and the higher 
than first-order motion distribution image generated from the k-th image. 

14. The apparatus according to claim 13, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k+ 1)-th image. 

15. An apparatus for estimating motion of a motion picture, 

the motion picture comprising a plurality of images, the apparatus comprising: 

a circuit (4) for estimating from a k-th image and a (k + 1)-th image a motion parameter corresponding to each 
pixel of the k-th image to generate a first-order motion distribution image corresponding to the k-th image, the 
first-order motion distribution image corresponding to the k-th image represented by the motion parameters 
corresponding to the pixels of the k-th image; 

a circuit (4) for estimating from at least the (k + 1)-th image a motion parameter corresponding to each pixel of 
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the (k + 1)-th image to generate a first-order motion distribution image corresponding to the (k + 1)-th image, 
the first-order motion distribution image corresponding to the (k + 1)-th image represented by the motion 
parameters corresponding to the pixels of the (k + 1)-th image; 

a circuit (5) for updating the first-order motion distribution image corresponding to the (k + 1)-th image to gen- 
erate a second-order motion distribution image corresponding to the (k + 1)-th image; 

means (6) for predicting, from the image second-order motion distribution image corresponding to the (k + 1)- 

th image, a third-order motion distribution image corresponding to the (k + 2)-th image; and 

a circuit (7) for interpolating the third-order motion distribution image corresponding to the (k + 2)-th image to 

generate a fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is an integer, 

and 

wherein the circuit for updating updates the first-order motion image distribution image corresponding to the (k 
+ 1)-th image using a higher than first-order motion distribution image generated from the k-th image. 

16. The apparatus according to claim 15, wherein the 

higher than first-order motion distribution image generated from the k-th image is a fourth-order motion distribution 
image corresponding to the (k + 1)-th image. 

17. The apparatus according to claim 15, wherein the 

higher than first-order motion distribution image generated from the k-th image is a fourth-order motion distribution 
image corresponding to the (k + l)-th image, wherein the circuit for predicting predicts from the first-order motion 
distribution image corresponding to the (k + 1)-th image a third-order motion distribution image corresponding to 
the k-th image; and 

the interpolating means interpolates the (k + 1 )-th third-order motion distribution image to generate the fourth-order 
motion distribution image corresponding to the (k + 1)-th image. 

18. The apparatus according to anyone of claims 15 to 17, 

wherein the circuit for predicting predicts a motion distribution image corresponding to the (k + 2)-th image from the 
second-order image corresponding to the (k + 1)-th image with an assumption that pixels move at a same speed 
as in the second-order image corresponding to the (k + 1)-th image. 

19. The apparatus according to anyone of claims 15 to 18, 

wherein the motion parameter includes a motion vector and an indication of reliability 

20. The apparatus according to claim 19, 

wherein the circuit (5) for updating combines, for each pixel, a motion vector representing the first-order motion dis- 
tribution image corresponding to the (k + 1)-th image and a motion vector representing a higher than first-order 
motion distribution image generated from the k-th image according to the reliability of the first-order and the higher 
than first-order motion distribution image generated from the k-th image. 

21 . The apparatus according to claim 20, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 

22. An apparatus for estimating motion of a motion picture, 

the motion picture comprising a plurality of images, the apparatus comprising: 

a first estimator for estimating from a k-th image and a (k + 1)-th image a motion parameter corresponding to 
each pixel of the k-th image to generate a first-order motion distribution image corresponding to the k-th image, 
the first-order motion distribution image corresponding to the k-th image represented by the motion parameters 
corresponding to the pixels of the k-th image; 

a second estimator for estimating from at least the (k + 1)-th image a motion parameter corresponding to each 
pixel of the (k + 1)-th image to generate a first-order motion distribution image corresponding to the (k + 1)-th 
image, the first-order motion distribution image corresponding to the (k + 1)-th image represented by the 
motion parameters corresponding to the pixels of the (k + 1)-th image; 

an updater (5) for updating the first-order motion distribution image corresponding to the (k + 1)-th image to 
generate a second-order motion distribution image corresponding to the (k + 1)-th image; 
a predictor (6) for predicting, from the image second-order motion distribution image corresponding to the (k + 
1)-th image, a third-order motion distribution image corresponding to the (k + 2)-th image; and 
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an interpolator (7) for interpolating the third-order motion distribution image corresponding to the (k + 2)-th 
image to generate a fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is 
an integer, and 

wherein the updater updates the first-order motion image distribution image corresponding to the (k + 1)-th 
image using a higher than first-order motion distribution image generated from the k-th image. 

23. The apparatus according to claim 22, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 

24. The apparatus according to claim 22, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image, 

wherein the predictor (6) predicts from the first-order motion distribution image corresponding to the k-th 
image a third-order motion distribution image corresponding to the (k + 1)-th image; and 

the interpolator (7) interpolates the (k + 1)-th third-order motion distribution image to generate the fourth-order 
motion distribution image corresponding to the (k + 1)-th image. 

25. The apparatus according to anyone of claims 22 to 24, 

wherein the predictor (6) predicts a motion distribution image corresponding to the (k + 2)-th image from the sec- 
ond-order image corresponding to the (k+ 1)-th image with an assumption that pixels move at a same speed as in 
the second-order image corresponding to the (k + 1 )-th image. 

26. The apparatus according to anyone of claims 22 to 25, 

wherein the motion parameter includes a motion vector and an indication of reliability. 

27. The apparatus according to claim 26, 

wherein the circuit (5) for updating combines, for each pixel, a motion vector representing the first-order motion dis- 
tribution image corresponding to the (k + 1)-th image and a motion vector representing a higher than first-order 
motion distribution image generated from the k-th image according to the reliability of the first-order and the higher 
than first-order motion distribution image generated from the k-th image. 

28. The apparatus according to claim 27, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k+ 1)-th image. 

29. A program storage device readable by a machine, 

tangibly embodying a program of instructions executable by the machine to perform method steps for estimating 
motion of a motion picture, the method steps comprising: 

estimating (4) from a k-th image and a (k + 1)-th image a motion parameter corresponding to each pixel of the 
k-th image to generate a first-order motion distribution image corresponding to the k-th image, the first-order 
motion distribution image corresponding to the k-th image represented by the motion parameters correspond- 
ing to the pixels of the k-th image; 

estimating (4) from at least the (k+ 1)-th image a motion parameter corresponding to each pixel of the (k + 1)- 
th image to generate a first-order motion distribution image corresponding to the (k + 1 )-th image, the first-order 
motion distribution image corresponding to the (k + 1)-th image represented by the motion parameters corre- 
sponding to the pixels of the (k + 1)-th image; 

updating (5) the first-order motion distribution image corresponding to the (k + 1)-th image to generate a sec- 
ond-order motion distribution image corresponding to the (k+ 1)-th image; 

predicting (6) , from the image second-order motion distribution image corresponding to the (k + 1)-th image, 
a third-order motion distribution image corresponding to the (k + 2)-th image; and 

interpolating (7) the third-order motion distribution image corresponding to the (k + 2)-th image to generate a 
fourth-order motion distribution image corresponding to the (k + 2)-th image, where k is an integer, and 
wherein the step of updating updates the first-order motion image distribution image corresponding to the (k + 
1)-th image using a higher than first-order motion distribution image generated from the k-th image. 

30. The program storage device according to claim 29, 
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wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 

31. The program storage device according to claim 29 

5 wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 

distribution image corresponding to the (k + 1)-th image, said method steps further comprising: 
predicting (6) from the first-order motion distribution image corresponding to the k-th image a third-order motion dis- 
tribution image corresponding to the (k + 1)-th image; and 

w interpolating (7) the (k + 1 )-th third-order motion distributionimage to generate the fourth-order motion distribu- 

tion image corresponding to the (k + 1)-th image. 

32. The program storage device to anyone ofclaims 29 to 31 , 

wherein the step of predicting predicts a motion distribution image corresponding to the (k + 2)-th image from the 
is second-order image corresponding to the (k + 1)-th image with an assumption that pixels move at a same speed 
as in the second-order image corresponding to the (k + 1)-th image. 

33. The program storage device according to claim 29, 

wherein the motion parameter includes a motion vector and an indication of reliability. 

20 

34. The program storage device according to claim 33, 

wherein the step of updating combines, for each pixel, a motion vector representing the first-order motion distribu- 
tion image corresponding to the (k + 1)-th image and a motion vector representing a higher than first-order motion 
distribution image generated from the k-th image according to the reliability of the first-order and the higher than 
25 first-order motion distribution image generated from the k-th image. 

35. The program storage device according to claim 34, 

wherein the higher than first-order motion distribution image generated from the k-th image is a fourth-order motion 
distribution image corresponding to the (k + 1)-th image. 

30 
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